Most of the current trajectory-based abnormal behavior detection algorithms do not consider the internal information of the trajectory, which might lead to a high false alarm rate. An abnormal behavior detection method based on trajectory segment using the topic model was presented. Firstly, the original trajectories were partitioned into trajectory segments according to turning angles. Secondly, the behavior characteristic information was extracted by quantifying the observations from these segments into different visual words. Then the time-space relationship among the trajectories was explored by Latent Dirichlet Allocation (LDA) model. Finally, the behavior pattern analysis and the abnormal behavior detection could be implemented by learning the corresponding generative topic model combined with the Bayesian theory. Simulation experiments of behavior pattern analysis and abnormal behavior detection were conducted on two video scenes, and different kinds of abnormal behavior patterns were detected. The experimental results show that, combining with trajectory segmentation, the proposed method can dig the internal behavior characteristic information to identify a variety of abnormal behavior patterns and improve the accuracy of abnormal behavior detection.